{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CTR模型训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 任务简介： \n",
    "广告点击率（Click-Through Rate Prediction, CTR）是互联网计算广告中的关键环节，预估准确性直接影响公司广告收入。机器学习技术可在计算广告中大展身手，Avazu通过程序化广告技术进行效果营销。本项目我们对Avazu提供的Kaggle竞赛数据进行移动CTR预估，其Kaggle竞赛网页为：https://www.kaggle.com/c/avazu-ctr-prediction。 \n",
    "\n",
    "属性：\n",
    "id: ad identifier （ID）\n",
    "click: 0/1 for non-click/click (是否被点击 0否、1是)\n",
    "hour: format is YYMMDDHH, so 14091123 means 23:00 on Sept. 11, 2014 UTC.（时间）\n",
    "C1 -- anonymized categorical variable（类型变量）\n",
    "banner_pos （广告位置）\n",
    "site_id   （站点ID）\n",
    "site_domain (站点领域)\n",
    "site_category  （站点类别）\n",
    "app_id   （APP_ID）\n",
    "app_domain (APP_领域)\n",
    "app_category （APP_类别）\n",
    "device_id (设备ID)\n",
    "device_ip (设备IP)\n",
    "device_model （设备模型）\n",
    "device_type （设备类型）\n",
    "device_conn_type （设备连接类型）\n",
    "C14-C21 -- anonymized categorical variables （（类型变量））"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#导包\n",
    "import numpy as np #矩阵操作\n",
    "import pandas as pd # 读取数据 sql操作\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#读取数据\n",
    "train = pd.read_csv('train_FE.csv')\n",
    "test = pd.read_csv('test_FE.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 100000 entries, 0 to 99999\n",
      "Columns: 120 entries, Unnamed: 0 to top_30_device_model\n",
      "dtypes: float64(1), int64(119)\n",
      "memory usage: 91.6 MB\n"
     ]
    }
   ],
   "source": [
    "train.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 100000 entries, 0 to 99999\n",
      "Columns: 119 entries, Unnamed: 0 to top_30_device_model\n",
      "dtypes: float64(1), int64(118)\n",
      "memory usage: 90.8 MB\n"
     ]
    }
   ],
   "source": [
    "test.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([u'Unnamed: 0', u'id', u'click', u'C1', u'banner_pos', u'device_type',\n",
       "       u'device_conn_type', u'C14', u'C15', u'C16',\n",
       "       ...\n",
       "       u'top_30_device_ip', u'top_10_device_model', u'top_25_device_model',\n",
       "       u'top_5_device_model', u'top_50_device_model', u'top_1_device_model',\n",
       "       u'top_2_device_model', u'top_15_device_model', u'top_20_device_model',\n",
       "       u'top_30_device_model'],\n",
       "      dtype='object', length=120)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([u'Unnamed: 0', u'id', u'C1', u'banner_pos', u'device_type',\n",
       "       u'device_conn_type', u'C14', u'C15', u'C16', u'C17',\n",
       "       ...\n",
       "       u'top_30_device_ip', u'top_10_device_model', u'top_25_device_model',\n",
       "       u'top_5_device_model', u'top_50_device_model', u'top_1_device_model',\n",
       "       u'top_2_device_model', u'top_15_device_model', u'top_20_device_model',\n",
       "       u'top_30_device_model'],\n",
       "      dtype='object', length=119)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_id = train['id']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_id = test['id']\n",
    "test = test.drop(['id'],axis=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train = train['click']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train = train.drop(['click','id'],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# #将数据分割训练数据与测试数据\n",
    "# from sklearn.cross_validation import  train_test_split\n",
    "\n",
    "# X_train,X_test,y_train,y_test = train_test_split(train,train_lable,random_state=33,test_size=0.25)\n",
    "\n",
    "# # 随机采样25%的数据构建测试样本，其余作为训练样本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 数据标准化\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# 分别初始化对特征和目标值的标准化器\n",
    "ss_X = StandardScaler()\n",
    "\n",
    "# 分别对训练和测试数据的特征以及目标值进行标准化处理\n",
    "X_train = ss_X.fit_transform(X_train)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 正则化的 Logistic Regression及参数调优\n",
    "logistic回归的需要调整超参数有：C（正则系数，一般在log域（取log后的值）均匀设置候选参数）和正则函数penalty（L2/L1） \n",
    "目标函数为：J = sum(logloss(f(xi), yi)) + C* penalty \n",
    "\n",
    "在sklearn框架下，不同学习器的参数调整步骤相同：\n",
    "设置候选参数集合\n",
    "调用GridSearchCV\n",
    "调用fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=2,\n",
       "       param_grid={'penalty': ['l1', 'l2'], 'C': [0.001, 0.01, 0.1, 1, 10, 100]},\n",
       "       pre_dispatch=6, refit=True, return_train_score=True,\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "penaltys = ['l1','l2']\n",
    "Cs = [0.001,0.01,0.1,1,10,100]\n",
    "tuned_parameters = dict(penalty = penaltys,C = Cs)\n",
    "lr_penalty = LogisticRegression()\n",
    "grid = GridSearchCV(lr_penalty,tuned_parameters,cv=5,scoring='neg_log_loss',return_train_score=True,n_jobs =2,pre_dispatch=6)\n",
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 0.1, 'penalty': 'l1'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.40969311522232083"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegressionCV(Cs=[0.001, 0.01, 0.1, 1, 10, 100], class_weight=None,\n",
       "           cv=5, dual=False, fit_intercept=True, intercept_scaling=1.0,\n",
       "           max_iter=100, multi_class='ovr', n_jobs=1, penalty='l1',\n",
       "           random_state=None, refit=True, scoring='neg_log_loss',\n",
       "           solver='saga', tol=0.0001, verbose=0)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "\n",
    "Cs = [0.001,0.01,0.1,1,10,100]\n",
    "\n",
    "# 大量样本、高维度（93），L1正则 --> 可选用saga优化求解器(0.19版本新功能)\n",
    "# LogisticRegressionCV比GridSearchCV快\n",
    "lrcv_L1 = LogisticRegressionCV(Cs=Cs, cv = 5, scoring='neg_log_loss', penalty='l1', solver='saga', multi_class='ovr')\n",
    "lrcv_L1.fit(X_train, y_train)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.        , -0.35254663,  0.01594767,  0.21646364, -0.01980474,\n",
       "        -0.25372047, -0.06774838,  0.23216982,  0.10934492,  0.01374575,\n",
       "         0.1919131 , -0.07474514, -0.11269949, -0.00305223,  0.00804516,\n",
       "        -0.01956932,  0.        ,  0.00546188,  0.00147939,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.09770634, -0.04401117,  0.06818657,\n",
       "        -0.00830614, -0.18871399,  0.32041582,  0.0032965 ,  0.05064363,\n",
       "         0.02420983, -0.0880051 , -0.01609394, -0.25392625,  0.01327111,\n",
       "         0.38300888, -0.02850841,  0.02161273, -0.02533491, -0.00205617,\n",
       "        -0.00556991,  0.        , -0.0060565 ,  0.00329313, -0.0060565 ,\n",
       "        -0.0060565 ,  0.05889536,  0.19189432, -0.1493458 , -0.07948467,\n",
       "         0.        ,  0.0221973 , -0.00777435,  0.02878499,  0.04630179,\n",
       "         0.05382676,  0.00845513, -0.04513441,  0.09624929,  0.01388041,\n",
       "        -0.07765612,  0.00830171, -0.04115939, -0.0306504 , -0.08828845,\n",
       "         0.02215932,  0.01558363, -0.23130632, -0.05653889,  0.165793  ,\n",
       "        -0.04650324,  0.23052506,  0.23052506,  0.2222726 , -0.00560896,\n",
       "         0.06719319,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        , -0.0142143 ,  0.        ,  0.        ,  0.        ,\n",
       "        -0.02531956,  0.        , -0.02697116,  0.        ,  0.06686217,\n",
       "        -0.05681576, -0.02531956, -0.02531956,  0.        , -0.0353467 ,\n",
       "         0.        ,  0.0424084 , -0.00669936, -0.02208818,  0.01436145,\n",
       "         0.02103596,  0.01580753, -0.01496817]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lrcv_L1.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1: array([[-0.42026101, -0.41150314, -0.41020668, -0.41016427, -0.41016843,\n",
       "         -0.41016904],\n",
       "        [-0.41814822, -0.40816006, -0.40648041, -0.40640638, -0.40640569,\n",
       "         -0.40640568],\n",
       "        [-0.42037594, -0.41089579, -0.40949811, -0.40961043, -0.40962948,\n",
       "         -0.40963139],\n",
       "        [-0.41980821, -0.41164695, -0.41096331, -0.4111449 , -0.41117775,\n",
       "         -0.41118105],\n",
       "        [-0.42038218, -0.41182746, -0.4113188 , -0.4114228 , -0.411437  ,\n",
       "         -0.41143844]])}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lrcv_L1.scores_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 通过基本模型的检测 无论是LR+L1 还是通过GridSearchCV 来检测 都是说最佳参数0.1以及使用L1正则然后logloss 是-0.40969311522232083；\n",
    "同时可以发现 当训练数据较大的时候 使用LRCV 会比GridSearchCV快 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.externals import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['lrcv_L1.m']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(grid,'grifSearch.m')\n",
    "joblib.dump(lrcv_L1,'lrcv_L1.m')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
